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Technologists  and  human  factors  practitioners  tend  to  approach  the  measurement  of 
situation  awareness  from  different  perspectives.  Technologists  compare  the  difference  between 
the  data  available  in  the  environment  with  what  has  been  detected  by  the  sensors  built  into  a 
system.  Human  factors  practitioners  focus  on  perception  and  cognition  to  the  exclusion  of  the 
technological  parts  of  the  system.  The  authors  propose  a  Dynamic  Model  of  Situated  Cognition 
and  use  it  as  a  framework  for  analyzing  both  the  technological  and  human  aspects  of  a  complex 
system.  They  employ  a  process  tracing  method  in  the  analysis  of  a  high  fidelity  military 
command  and  control  (C2)  simulation.  Their  results  indicate  that  the  model  and  the  process 
tracing  method  are  effective  ways  in  which  to  investigate  the  development  of  situated  cognition  in 
complex  systems.  In  addition,  their  results  have  important  implications  for  designers  of  software, 
hardware,  and  training  systems. 


INTRODUCTION 

Complex  cognitive  systems  couple  humans 
with  machines  for  the  purpose  of  accomplishing  a 
specific  goal.  It  is  often  the  case  that  human  factors 
practitioners  focus  their  attention  on  the  humans  in 
the  system  while  designers  and  engineers  tend  to 
focus  on  the  technological  aspects  of  the  system. 
Human  factors  practitioners  tend  not  to  study  or 
evaluate  the  technological  aspects  of  a  system 
because  they  may  lack  the  necessary  access  or 
expertise.  Systems  analysts  and  engineers  may  try 
to  reduce  human  behavior  to  a  series  of  stochastic 
equations  which  may  give  the  appearance  of 
accuracy  and  precision  but  ultimately  will  miss  the 
complexity,  creativity,  and  variability  of  perception 
and  cognition. 

While  systems  analysts  and  engineers 
appear  to  have  made  progress  in  describing  and 
evaluating  the  activity  within  the  technological 
aspects  of  a  complex  system,  human  factors 
practitioners  continue  to  wrestle  with  evaluating 
human  cognitive  processing  in  such  systems. 
Further,  many  practitioners  evaluate  cognitive 
activities  as  if  they  were  states  rather  than 
processes.  They  talk  about  cognition  at  a  particular 
point  in  time  in  terms  of  percentages,  comparing 


what  was  reported  by  the  decision  maker  to  what 
should  have  been  known. 

An  alternative  view,  and  one  which  is  held 
by  the  authors  of  this  paper,  is  that  it  is  more 
appropriate  to  consider  and  evaluate  cognitive 
activities  as  a  process.  For  example,  how  much 
situation  awareness  (SA)  a  decision  maker  has  at 
any  point  is  important,  but  even  more  important  is 
how  the  SA  evolved  over  time,  as  well  as  when  and 
how  the  SA  deviated  from  ground  truth.  In  this 
paper,  the  authors  provide  a  new  model  -  a  Dynamic 
Model  of  Situated  Cognition  -  that  integrates  human 
cognition  with  the  technological  systems  that 
provide  data  and  information  to  the  decision  maker. 

The  authors  also  present  a  process  tracing 
approach  for  analyzing  cognitive  processes  across 
the  human  -  machine  system.  The  process  tracing 
method  combines  multiple  measures  that  permit 
researchers  to  chronicle  the  changes  in  cognitive 
activities  as  events  unfold,  to  highlight  key  events, 
and  to  identify  points  at  which  the  understanding  of 
decision  makers  deviates  from  ground  truth  and 
why  those  deviations  occur.  The  authors  applied 
both  the  Dynamic  Model  of  Situated  Cognition  and 
the  process  tracing  method  to  guide  the 
investigation  of  human  -  system  performance 
during  a  recent  military  command  and  control  (C2) 
high  fidelity  simulation. 
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In  recent  years,  human  factors  practitioners 
have  developed  a  variety  of  methods  to  assess 
human  cognition  and,  in  particular,  situation 
awareness  (see  Gawron,  2000).  The  methods  vary 
in  their  degree  of  subjectivity,  rigor,  and 
intrusiveness.  One  method  simply  asks  decision 
makers  to  rate  their  situation  awareness  by  referring 
to  a  behaviorally  anchored  scale.  Other  methods 
attempt  to  compare  the  data  available  in  the 
environment  with  the  decision  maker’s  perception, 
comprehension,  and  projection.  These  comparisons 
will  result  in  a  more  accurate  assessment  of  a 
decision  maker’s  mental  model  of  the  battlefield 
than  the  methods  proposed  by  technologists. 
However,  they  are  still  problematic.  One  typical 
method,  Situation  Awareness  Global  Assessment 
Technique  (SAG AT),  involves  stopping  the  flow  of 
activity  -  to  the  dismay  of  the  decision  makers  -  at 
multiple  points  in  time  and  asking  them  a  series  of 
questions.  The  questions  normally  are  linked  to  the 
various  levels  of  situation  awareness  (perception, 
comprehension,  projection). 

There  are  several  sources  of  subjectivity  in 
SAGAT.  Developing  the  questions  requires  a  keen 
understanding  of  both  the  command  and  control 
environment  and  the  levels  of  situation  awareness. 
Several  questions  must  be  developed  for  each  level 
of  situation  awareness  in  order  to  conduct  statistical 
analysis  on  the  responses.  The  questions  posed  for 
each  point  in  time  must  be  different  to  prevent 
decision  makers  from  artificially  directing  their 
attention  in  anticipation  of  the  probe.  Yet,  the 
different  questions  within  each  level  must  be 
equivalent  in  difficulty  or  the  results  will  be  invalid. 
Scoring  can  also  be  challenging,  whether  the 
questions  are  verbal  (i.e.,  short  answer  or  fill-in-the 
blank)  or  graphical  (i.e.,  placing  unit  symbols  on  a 
map).  The  researchers  who  score  the  questions 
must  have  access  to  the  data  in  the  environment  to 
determine  the  correct  answers  at  the  time  of  the 
probe.  Also,  the  scoring  of  the  researchers  must  be 
calibrated  to  ensure  their  subjective  judgments  are 
similar. 

SAGAT  and  related  methods  of  analysis  are 
heavily  dependent  upon  subject  matter  experts 
(SMEs).  Among  the  roles  filled  by  SMEs  are  the 
following.  SMEs: 

•  are  intimately  familiar  with  the  domain  being 
studied. 


•  serve  as  a  guide  for  researchers  into  the  domain 
of  practice. 

•  can  facilitate  access  to  data. 

•  are  able  to  formulate  appropriate  SAGAT 
questions. 

•  can  assist  researchers  in  understanding  the 
behavioral  responses  of  the  study  participants 
within  the  context  of  that  domain. 

When  used  by  themselves  to  assess  human 
awareness  and  cognition  in  context,  methods  such 
as  SAGAT  still  miss  the  mark.  The  most  rigorous 
will  compare  the  data  available  in  the  environment 
with  the  perception,  comprehension,  and  projection. 
However,  they  do  not  account  for  the  technological 
processes  that  contribute  to  human  cognition.  A 
possible  reason  for  this  shortcoming  is  that  the 
models  upon  which  these  methods  are  based  tend  to 
be  parochial.  Methods  that  measure  technological 
awareness  are  based  on  models  that  emphasize  the 
software  and  hardware  portions  of  the  system. 
Methods  that  measure  perceptual/cognitive 
processes  are  based  on  models  that  emphasize  the 
human  portions  of  the  system.  The  model  described 
below  considers  both  the  technological  and  human 
components  of  the  system  and,  therefore,  serves  as  a 
more  viable  representation  from  which  to  develop 
methods  for  assessing  awareness  and  situated 
cognition. 

A  DYNAMIC  MODEL  OF  SITUTATED 
COGNITION 

Figure  1  depicts  the  Dynamic  Model  of 
Situated  Cognition.  The  description  that  follows 
uses  a  military  C2  context  to  describe  the  model, 
but  the  model  is  equally  relevant  in  any  domain  in 
which  humans  and  machines  interact  with  one 
another  to  achieve  a  desired  end  state.  Oval  1 
contains  all  data  available  in  the  environment.  The 
blue  rectangles  and  red  diamonds  represent  friendly 
and  enemy  data  elements.  (For  simplicity,  the 
figure  does  not  include  data  elements  that  represent 
other  physical  characteristics  of  the  environment 
such  as  non-combatants,  terrain,  and  weather.) 

Oval  2  contains  only  those  data  in  Oval  1  that  are 
able  to  be  detected  by  the  sensors  in  the  system. 

Oval  2  is  smaller  than  Oval  1  because  in  most 
systems,  sensors  are  insufficient,  in  quantity  or 
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Figure  1.  A  Dynamic  Model  of  Situated  Cognition. 


capability,  to  detect  all  data  in  the  environment.  In 
highly  dynamic  environments  such  as  military  C2, 
decision  makers  often  are  unaware  of  which 
portions  of  the  system  are  ‘covered’  by  sensors  and 
which  are  not. 

Oval  3  depicts  those  data  that  are  presented 
to  the  decision  maker,  often  on  a  visual  display,  but 
sometimes  using  auditory  and  haptic  modalities. 
Although  it  may  be  possible  for  all  data  in  Oval  2  to 
be  presented,  decision  makers  tailor  their  displays  to 
reduce  clutter.  The  display  is,  in  essence,  a  keyhole 
through  which  the  decision  maker  peers  to  gain 
access  to  the  vast  amount  of  data  stored  in  the 
system.  Ovals  1,  2,  and  3  represent  the 
technological  part  of  the  human-machine  system. 

Three  distinct  lenses  are  depicted  in  Figure 
1 .  Although  the  informational  elements  are  the 
same  in  each  lens,  the  placement  of  the  lenses  in  the 
model  suggests  that  different  functions  are 
performed  by  each  lens.  As  is  the  case  with  the 


human  visual  lens,  perceptual  distortions  may  result 
from  asymmetries  and  flaws  in  the  refining  process. 

There  are  at  least  four  classes  of  information 
embedded  in  the  lenses  that  influence  how  decision 
makers  perceive,  comprehend  and  make  predictions 
about  activities  on  the  battlefield.  The  local 
situation  influences  the  data  to  which  a  decision 
maker  will  attend.  The  operation  order  (OPORD) 
represents  the  specific  plan  a  unit  is  attempting  to 
execute.  Doctrine  represents  broad  guidelines  to 
which  decision  makers  may  refer  if  the  OPORD  is 
underspecified.  Experience  refers  to  previous 
activities  in  which  a  decision  maker  has  engaged 
that  are  similar  to  the  current  situation.  Decision 
makers  rely  (either  consciously  or  unconsciously) 
on  these  previous  experiences  to  influence  how  they 
direct  their  attention.  Data  useful  in  previous 
situations  may  prove  useful  in  the  current  situation. 
Together,  these  four  classes  of  information 
influence  what  is  perceived  by  the  decision  maker. 


Lens  A,  the  lens  between  Ovals  3  and  4, 
directs  attention  to  selected  incoming  stimuli.  Lens 
A  will  focus  or  direct  attention  to  a  subset  of  the 
data  available  in  Oval  3.  These  data  will  be 
perceived  in  Oval  4  by  the  decision  maker. 
Alternatively,  data  which  would  not  normally  be 
focused  on  by  the  decision  maker  but  that  are 
presented  in  a  highly  salient  manner  (e.g.,  bright  or 
loud)  may  intrude  upon  the  decision  maker’s 
cognitive  processes  and  also  will  be  perceived. 

The  data  perceived  by  the  decision  maker 
are  interpreted,  organized,  and  integrated  based  on 
the  contents  of  lens  B,  resulting  in  comprehension 
(Oval  5).  The  lens  between  Ovals  5  and  6,  Lens  C, 
guides  the  process  of  extrapolating  current 
information  into  predictions  about  the  future.  Ovals 
4,  5,  &  6  and  lenses  A,  B,  and  C  comprise  the 
human  part  of  the  system. 

In  general,  it  can  be  thought  that  data  flow 
from  left  to  right.  Although  not  depicted  in  Figure 
1,  there  are  feedback  loops  that  flow  from  right  to 
left.  Output  from  Ovals  4,  5,  and  6  can  influence 
the  contents  of  preceding  ovals.  For  example,  if  a 
decision  maker  perceives  that  an  enemy  vehicle  has 
been  detected  (Oval  4)  and  comprehends  it  is  a 
threat  (Oval  5),  then  a  decision  to  deploy  a  sensor 
could  change  what  is  detected  in  the  environment 
(Oval  2),  which  could  lead  to  changes  in  Ovals  3  - 
6,  as  well  as  the  lenses.  Output  from  Oval  5 
(comprehension)  and  Oval  6  (projection)  can  also 
affect  the  lenses.  For  example,  understanding  the 
outcome  of  a  battle  in  which  a  particular  maneuver 
was  used  could  change  the  contents  and  the  contour 
of  the  lens  because  the  event  has  added  to  the 
experience  base  of  the  decision  maker. 

C2  SIMULATION 

Participants 

Participants  in  the  simulation  consisted  of 
four  male  active  duty  Army  soldiers  who  occupied  a 
high  fidelity  C2  mock-up.  Three  were  officers  and 
one  was  a  non-commissioned  officer.  The 
simulation  also  included  active  duty  and  retired 
Army  officers  who  role  played  the  higher 
headquarters  and  the  enemy. 


Equipment 

Each  participant  had  his  own  workstation  in 
the  mock-up,  which  consisted  of  two  computer 
monitors  and  a  communications  console.  The 
mock-up  also  had  a  large  screen  centrally  located, 
which  any  of  the  participants  could  use  to  display 
what  was  on  either  of  their  workstation  monitors. 

Procedures 

The  participants  were  given  two  weeks  of 
training  prior  to  the  actual  data  collection.  During 
these  two  weeks,  the  participants  became 
acquainted  with  one  another,  were  trained  on  the  C2 
software,  determined  the  roles  they  would  play,  and 
developed  their  tactics,  techniques,  and  procedures. 

There  were  six  90-minute  sessions  that  were 
conducted  over  four  days.  Aside  from  a  software 
constraint  that  limited  the  size  of  the  maneuver  area, 
the  simulation  permitted  friendly  and  enemy 
participants  to  exercise  free  play.  Each  90-minute 
session  was  preceded  by  a  period  during  which  the 
participants  received  their  mission  from  higher 
headquarters,  analyzed  their  intelligence,  developed 
a  plan,  and  emplaced  friendly  units  in  their  initial 
positions. 

The  authors  used  the  Dynamic  Model  of 
Situated  Cognition  as  the  basis  for  tracing  the  flow 
of  data  and  the  development  of  awareness  over 
time.  This  process  tracing  method  relied  on 
multiple  data  sources  to  construct  a  complete  and 
accurate  story  of  how  the  events  unfolded  (see 
Woods,  1993)  across  both  the  technological  and  the 
perceptual/cognitive  components  of  the  C2  system. 
Sources  of  data  included: 

•  Videotapes  of  the  eight  computer  terminals  (two 
per  participant). 

•  Voice  transcripts. 

•  Database  queries. 

•  Heart  rate  variability  monitors  (worn  by  each 
participant). 

•  Geographical  Recall  and  Analysis  of  Data  in  the 
Environment  (GRADE).  At  selected  times 
through  each  session,  participants  were  asked  to 
sketch  on  a  map  what  was  happening  on  the 
portion  of  the  battlefield  on  which  they  were 
focused  and  to  predict  what  they  thought  would 
be  happening  thirty  minutes  into  the  future. 


•  Situation  Awareness  Rating  Technique  (SART). 
Participants  rated  their  awareness  along  three 
dimensions:  demand,  supply,  and 
understanding. 

•  Retrospective  interviews. 

Results  of  the  study  support  using  the 
Dynamic  Model  of  Situated  Cognition  as  a 
framework  for  analysis  of  the  events  in  the  C2 
simulation.  The  authors  were  able  to  construct 
detailed  chronologies  by  combining  the  various  data 
sources.  These  chronologies  describe  the 
technological  awareness  (the  data  that  were 
available  in  the  environment,  detected  by  sensors, 
and  displayed  on  the  workstation  screens)  and  the 
human  awareness  (perception,  comprehension,  and 
projection).  The  chronologies  demonstrate  the 
influences  the  lenses  had  in  directing  attention  and 
in  interpreting  the  data.  The  chronologies  also 
showed  how  feedback  loops  influenced  the  lenses 
as  well  as  shaped  the  content  of  the  ovals  in  the 
model. 

In  addition  to  the  detailed  chronologies  that 
resulted  from  adopting  the  Dynamic  Model  of 
Situated  Cognition  as  an  analytical  perspective, 
many  other  important  findings  emerged  .  Some  of 
these  findings  include  the  following. 

•  Collaboration.  The  C2  system  provided  many 
ways  in  which  the  participants  could 
collaborate.  In  spite  of  all  the  tools,  participants 
often  were  unaware  of  what  the  other 
participants  were  doing. 

•  Data  Overload.  The  C2  software  can  easily 
overload  decision  makers  with  data.  Designers 
understood  this  and  provided  users  the  ability  to 
tailor  their  workstations  in  a  manner  that 
reduced  screen  clutter.  During  one  90-minute 
session  9600  alerts  were  generated  by  the 
software.  However,  the  vast  majority  of  the 
alerts  never  were  displayed  because  the 
participants  had  configured  their  workstations  to 
filter  them  out.  As  a  result,  in  some  cases, 
important  data  detected  by  sensors  were  never 
viewed  by  the  participants. 

•  Attention  Shifting.  An  analysis  of  a  13  minute 
period  of  a  session  revealed  that  one  participant 
shifted  his  attention  26  times  for  an  average  of 
once  every  30  seconds.  During  a  16  minute 
period  of  another  session  the  same  participant 


shifted  his  attention  60  times  for  an  average  of 
once  every  16  seconds.  The  frequency  of 
attention  shifts  suggests  that  the  level  of 
processing  performed  by  the  participant  was 
relatively  shallow. 

•  Redundant/Repetitive  Activities.  Participants 
were  observed  to  perform  the  same  behavior 
multiple  times.  Participants  also  engaged  in 
redundant  behaviors,  duplicating  the  efforts  of 
other  participants.  These  behaviors  suggest  the 
possibility  that  there  may  be  some  issues  of 
display  design,  workload  distribution,  and 
coordination  that  need  to  be  resolved  in  order  to 
improve  performance  within  the  C2  system. 

The  study  conducted  by  the  authors 
demonstrates  the  utility  of  using  the  Dynamic 
Model  of  Situated  Cognition  as  a  framework  for 
examining  the  development  of  cognition  in  context. 
Employing  a  process  tracing  approach  to  the 
analysis  of  complex  systems  facilitates  the  linkage 
of  technological  and  human  awareness,  which, 
heretofore  have  been  treated  separately.  By 
collecting  data  at  each  stage  in  the  model,  a  detailed 
chronology  of  events  is  created  which, 
when  analyzed,  provides  invaluable  insights  into 
how  cognition  evolved.  More  importantly,  this 
approach  facilitates  the  understanding  of  when, 
where,  and  why  the  situated  cognition  of  the 
decision  makers  differs  from  reality.  These  findings 
will  lead  to  recommendations  on  the  design  of 
software,  hardware,  and  training  systems  that  will 
truly  benefit  future  users  of  complex  cognitive 
systems. 
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